Expectation Maximisation in Python: Coin Toss Example
Expectation Maximisation with Python : Coin Toss¶
This notebook implements the example, I consider a classic for understanding Expectation Maximisation.
See: http://www.nature.com/nbt/journal/v26/n8/full/nbt1406.html
Notations:
\begin{align*} \theta_A &= \text{Probability of a Heads showing up given the coin tossed is A}\\ \theta_B &= \text{Probability of a Heads showing up given the coin tossed is B}\\ \end{align*}Entropy and Uniform Distribution
To show \(\sum p_i\log(p_i) = p\)
Consider \(H = -\sum p_i log(p_i) = log(n)\)
Consider weighted AM-GM for \(p_i\):
Take log both sides:
A frequent inequality
Consider \(f(x)=\ln(x)-\frac{x-1}{x}\)
\(f'(x) = \frac{1}{x} - \frac{1}{x^2} = \frac{x-1}{x^2}\)
Now consider the following two cases:
Case A: \(0 < x \leq 1\) and Case B: \(1 < x < \infty\) ...
read moreFunctions of independent random variables are independent
A textbook problem
Given \(X\), \(Y\) are two independent random variables, show that functions \(g(X)\) and \(h(Y)\) are independent too
Short Proof
Never took a course on measure theory, so avoiding that route:
Let.
\(R = g(X)\)
\(S = h(Y)\)
Also define,
\(A_r = \{x: g(X)\leq r\}\) ...
read moreBernoulli random varlable and UMVUE
Problem
Let \(X_1,...,X_n\) random sample \(X\)~\(Bernoulli(p)\). For \(n\geq 4\) show that the product \(X_1X_2X_3X_4\) is a unbiased estimator for \(p^4\), and use this fact for find the best unbiased estimator of \(p^4\)
Solution
As posted on stackexchange
[ToDo: Add variance, prove \(E[\phi(T)]=0\) ...
read moreDistribution of one random variable less than other
Problem
\(P(Y<X)\) for any two independent random variables \(X,Y\)
Solution
We just follow the definition:
\(P(Y<X) = \int_{-\infty}^{\infty}f_X(x)dx \int_{-\infty}^{x}f_Y(y)dy\)
\(P(Y <X) = \int F_X(y)f_X(x)dx\)
read more[Proof]Weighted AM-GM
We make use of Jensen’s Inequality and the fact that \(log(x)\) is a concave function:
For concave f: \(f(\frac{\sum a_ix_i}{\sum a_i}) \geq \frac{\sum a_i f(x_i)}{\sum a_i}\)
\(f=log(X)\)
\(log(\frac{\sum a_ix_i}{\sum a_i}) \geq \frac{\sum a_i log(x_i)}{\sum a_i}\) ...
read moreExpected number of pairings | Jones and Smith Family
Problem
4 members of Smith family and 4 of Jones family are pooled to form all possible pairs, all equally likely. Find \(N\), the number of smiths which have a smith partner.
Solution
\(X= \sum_{i=1}^4 I_i\)
Where \(I_i=1\) iff \(i^{th}\) meber of smith family has ...
read morePoisson Demysitified
Learnt it the hard way. This one is more of a scratch pad, inspired from the previously discussed problme on estimating the size of restriction fragments.
Notations:
\(N_t\)= Number of events that have happened in the \(t^{th}\) time interval
\(T\) = Time taken till the first arrival occurs(or the ...
read moreProve: No UMVUE exists for $\frac{1}{\theta}$ for Poisson Distribution.
Problem
Proof: No UMVUE exists for \(\frac{1}{\theta}\) for Poisson Distribution.
Solution
http://stats.stackexchange.com/a/152660/11668
Observation 1: \(\sum X_i\) is complete and sufficient statistic for \(\theta\)
Observation 2: \(\sum X_i \sim Poisson(n\theta)\)
We need to look for an unbiased estimator of \(\frac{1}{\theta}\) ...
read moreRestriction Fragments, MLE, and Mixed Random Variables
Let \(X_1, X_2, \dots , X_n\) be the lengths of \(n\) restriction fragments. Suppose that a biotechnique can measure fragment lengths accurately up to a given length c. That is, if \(X_i < c\), then the technique gives correct value \(X_i\) Show that MLE of \(\lambda = \frac{n-T_n}{S_n+cT_n}\) where \(S_n= \sum_{j=1}^n X_jI(X_j < c)\) ...
read moreRestriction Fragments Size Distribution
Given
The number of restriction sites in a fragment of length \(t\) are distributed as:
Find the distribution of length of restriction fragments
Solution
Intuition: If I have a restriction site at some location \(z\) and want it to ...
read moreConvolution Demysitifed
Problem
Given \(f(x) = \frac{x}{2}\) for \(0 \leq x \leq 2\). Find the pdf of \(x_1+x_2\) for \(x_1,x_2\) which are i.i.d.
Wrong solution
Thus, blindly,
\(f_S(s) = \int_{0}^s \frac{x(s-x)}{4}dx = \frac{s^3}{24}\) ...
read moreMATH 501 Project
Introduction
Model of a two state reparable system:
Runs in flips of a coin
This problem happened to be in one of the screening examinations and is my favorite because it demonstrates an application of indicator variables
Problem
A run is defined as maximal subsequence of consecutive tosses all having the same outcome. So HHHTHHTTH has 5 runs.(HHH,T,HH,TT,H). Let ...
read moreSVD v/s MDS v/s PCA
Principle Component Analysis(PCA) is a relatively more famous than Singular Value Decomposition(SVD) or Multidimensional Scaling(MDS). When I was introduced to the latter two, I was utterly confused trying to figure out what goes in where.
SVD
Let \(X_{mxn}\) data matrix. For an easy to relate example ...
read moreMultidimensional Scaling
MDS is a statistical technique to visualize dissimilarity between points. The distances between two pointsin n-dimensions are visualized in 2 dimensions such that it represents the distance in n-dimensions as far as possible.
It is important to note that, for MDS, we start of with a ‘distance’ matrix and not ...
read moreGumbel distribution expectation
What
If \(X_1,..,X_n\) is a random sample with density \(f(x;\theta)=e^{-(x-\theta)}e^{-e^{-(x-\theta)}}\) (\(x \in\mathbb{R}\)) and \(-\infty<\theta<\infty\), \(\quad\)i) Find the estimator of \(\theta\)
Solution
First let’s confirm if \(f(x;\theta)=e^{-(x-\theta)}e^{-e^{-(x-\theta)}}\) ...
read moreBA = AB = I
To Prove: If
and
such that AB=I, then BA=I
Reason: Rank(AB)
min(Rank A, Rank B)
so B is a full rank matrix. Now consider B=BI
B-B(AB)=0
B-(BA ...
read moreL2 norm and spectral radius of symmetric matrix
L2 norm and spectral radius for
L2 inducded norm:
NOTE: L2 norm is not the same as Frobenius norm. L2 norm is an “induced vector” norm, Frobenius is a “matrix” norm.
Induced norm:
defined as ||A|| =
Note how ||Ax|| is a vector norm ...
read morePositive definite second derivative
read moreLet \(f:\mathbb{R}\to\mathbb{R}\) be a twice-differentiable function, and let \(f\)’s second derivative be continuous. Let \(f\) be convex with the following definition of convexity: for any \(a<b \in \mathbb{R}\):
$$f\left(\frac{a+b}{2}\right) \leq \frac{f(a)+f(b)}{2}$$ ...
Proof for triangle inequality for case $x+y<0$
\(-|x| \leq x \leq |x|\) and \(-|y| \leq y \leq |y|\) \(\implies\) \(-|x|-|y| \leq x+y \leq |x|+|y|\) \(\implies\) \(|x+y| \leq |x| +|y|\) for any real \(x,y\)
The last implication comes from the fact: \(|x| \leq a \leftrightarrow -a \leq x \leq a\) for some \(a \geq 0\) ...
read more